Method and apparatus for generating meta data of content

ABSTRACT

A method of providing content includes receiving emotional information in relation to the content; generating information about a representative type of emotion associated with the emotional information received from the client; updating an Electronic Program Guide associated with the content or a header of the content using the information about the representative type of emotion; and providing the content with the updated Electronic Program Guide or the updated header to the user.

CROSS-REFERENCE TO RELATED PATENT APPLICATION

This is a continuation of application Ser. No. 13/108,357 filed May 16,2011, which is a continuation of application Ser. No. 11/730,227 filedMar. 30, 2007 (now U.S. Pat. No. 7,953,254) which claims priority fromKorean Patent Application No. 10-2006-0105270, filed on Oct. 27, 2006,in the Korean Intellectual Property Office. The entire disclosures ofthe prior applications are considered part of the disclosure of theaccompanying continuation application and are hereby incorporated byreference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Methods and apparatuses consistent with the present invention relate tometa data of content, and more particularly, to generating emotionalinformation including a user's impressions in relation to multimediacontent or meta data regarding the emotional information.

2. Description of the Related Art

On-line services which provide multimedia content have been becomewidely popular. However, feedback regarding a user's emotions, feelings,etc. after he or she appreciates content depends on the user's voluntaryprovision of information. Accordingly, it is difficult to acquire theuser's emotional information after the user has appreciated the content.Although the user provides his or her emotional information, theaccuracy or objectivity of the information cannot be ensured.

SUMMARY OF THE INVENTION

The present invention provides a method and apparatus for automaticallyacquiring a user's emotional information in relation to content which heor she is appreciating, and reflecting the emotional information in metainformation of the content, and a computer-readable recording mediumstoring the method.

According to an aspect of the present invention, there is provided amethod of generating meta data of content, including: receivingemotional information in relation to the content from at least oneclient system which receives and reproduces the content; generating metadata for an emotion using the emotional information; and coupling themeta data for the emotion to the content.

The generating the meta data for the emotion includes determining arepresentative type of emotion representing the content.

The generating the meta data for the emotion includes determining arepresentative type of emotion representing each scene of the content.

The generating the meta data for the emotion includes determining arepresentative type of emotion for each predetermined time period duringreproduction of the content.

The coupling the meta data for the emotion to the content includesadding the meta data for the emotion to existing meta data of thecontent.

The adding the meta data for the emotion includes, if the content is abroadcast program, adding the meta data for the emotion to an electronicprogram guide (EPG) for the broadcast program.

The adding the meta data for the emotion includes adding the meta datafor the emotion to header information of the content.

The method further includes, before providing the content to the atleast one client system, adding information about a time required foracquiring a user image to the meta data of the content, to allow the atleast one client system to acquire the emotional information.

According to another aspect of the present invention, there is provideda method of generating emotional information in relation to content,including: acquiring a user image while the content is reproduced;extracting facial information from the user image; generating theemotional information in relation to the content by comparing the facialinformation with facial expression template information; andtransmitting the emotional information to a server which provides thecontent.

The acquiring the user image includes acquiring the user image whenevereach scene of the content is reproduced.

The acquiring the user image includes receiving at least two user imagesfrom an image input unit while the content is reproduced.

The generating the emotional information includes generatingrepresentative emotional information representing emotional informationgenerated from the at least two user images.

The acquiring the user image includes acquiring the user image at a timecorresponding to an emotional information acquiring time included in themeta data of the content.

The generating the emotional information includes, if a plurality ofpieces of the facial information are extracted from the user image,generating emotional information corresponding to each piece of thefacial information.

The generating the emotional information includes generating emotionalinformation including a type and strength of an emotion when the userimage is acquired, with reference to the facial expression templateinformation.

According to another aspect of the present invention, there is providedan apparatus for generating meta data of content, including: acommunication unit which receives emotional information in relation tothe content from at least one client system which receives andreproduces the content; and a meta data generating unit which generatesmeta data for an emotion using the emotional information, and couplesthe meta data for the emotion to the content.

According to another aspect of the present invention, there is providedan apparatus for generating emotional information in relation tocontent, including: an image input unit which acquires a user imagewhile the content is reproduced; a facial image extraction unit whichextracts facial information from the user image; an emotionalinformation generating unit which compares the facial information withfacial expression template information, and generates the emotionalinformation in relation to the content based on a result of thecomparison; and a communication unit which transmits the emotionalinformation to a server which provides the content.

The response of audiences which watch contents, such as movies, music,or moving pictures, appears by their expressions while appreciating thecontents. Particularly, whether a corresponding content producerstimulates a user's emotion according to his or her intention can beobjectively recognized by analyzing the user's expressions. The presentinvention generates emotional information using the expressioninformation of a user which uses content, and provides the emotionalinformation to the corresponding content producer, thereby allowing theuser's emotional information to be used as meta data of thecorresponding content. Hereinafter, a case where content, such as moviesor moving pictures, is reproduced is illustrated, however, the presentinvention is not limited to this. The present invention can be alsoapplied to different types of contents.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects of the present invention will become moreapparent by describing in detail exemplary embodiments thereof withreference to the attached drawings in which:

FIG. 1 is a block diagram of an emotional information generatingapparatus according to an exemplary embodiment of the present invention;

FIG. 2 is a flowchart of an emotional information generating methodaccording to an exemplary embodiment of the present invention;

FIG. 3 is a view illustrating facial expression template informationaccording to an exemplary embodiment of the present invention;

FIG. 4 is a table in which the types of emotional information arelisted, according to an exemplary embodiment of the present invention;

FIG. 5 is a view illustrating emotional information generated accordingto an exemplary embodiment of the present invention;

FIG. 6 is a flowchart of an emotional information generating methodaccording to another exemplary embodiment of the present invention;

FIG. 7 is a view explaining a relationship between a content providerand a content receiver according to an exemplary embodiment of thepresent invention; and

FIG. 8 is a flowchart illustrating a meta data generating methodaccording to an exemplary embodiment of the present invention.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS OF THE INVENTION

Hereinafter, the present invention will be described in detail byexplaining exemplary embodiments of the invention with reference to theattached drawings.

FIG. 1 is a block diagram of an emotional information generatingapparatus according to an exemplary embodiment of the present invention.

Referring to FIG. 1, the emotional information generating apparatus 100includes an image input unit 110 for acquiring images of a user (users)photographed by a camera, etc., a storage unit 130 for storing facialexpression template information 132, in which people's various facialexpressions are created as templates, and user's emotional information134, and a control unit 120 including a facial image extraction unit122, a facial image normalization unit 124, and an emotional informationgenerating unit 126, and controlling different components to generatethe user's emotional information while content is being reproduced.

The image input unit 110 receives a user image obtained by using acamera, etc. to photograph a user appreciating content. The user imageis acquired at a predetermined time. According to various methods, twoor more user images may be acquired for each item of content, and theuser images may be acquired periodically in a unit of a second, aminute, etc., or whenever a scene of the content changes. Thepredetermined time for acquiring the user image can be recognized, byinterpreting time information pre-stored in the emotional informationgenerating apparatus 100, time information received from a contentprovider, or emotional information acquirement time information includedin meta data of the content.

The facial image extraction unit 122 extracts facial image informationfrom the user image. In order to extract the facial image informationfrom the user image, a Gabor filter response can be used. In the casewhere a plurality of facial images are extracted from a plurality ofuser images, emotional information corresponding to each facial image isgenerated. The facial image normalization unit 124 adjusts the extractedfacial images according to the sizes of templates included in the facialexpression template information 132.

The emotional information generating unit 126 determines the user'sfacial expression by template matching, and generates and outputsemotional information according to the user's facial expression. If aplurality of facial images are extracted from the user images, emotionalinformation including the number of people and individual facialexpression information is generated. If two or more user images areacquired and then two or more facial expression images for the same userare acquired, a plurality of emotional information is generated. In thiscase, representative emotional information of the plurality of emotionalinformation may be generated. At this time, the type of emotion whichthe user most frequently has from among the plurality of emotionalinformation can be determined as a representative type of emotion.However, the present invention is not limited to this, and variousmethods can be used according to applications. The emotional informationis used as meta data of the corresponding content, and can bestatistically processed or integrated by various methods. For example,data including a representative type of emotion for each scene of thecontent, or data including individual emotional information is output.Also, the emotional information can further include informationregarding the strength of emotions as well as the types of emotions.

FIG. 2 is a flowchart of an emotional information generating methodaccording to an exemplary embodiment of the present invention.

Referring to FIG. 2, first, content is reproduced (operation 202). Whilethe content is reproduced, a user image is acquired at a predeterminedtime (operation 204). Then, only a facial image part is extracted fromthe acquired user image, so that facial information is generated(operation 206). The facial information is compared with facialexpression template information stored in the storage unit 130 to obtaina matched facial expression, and then emotional information for thematched facial expression is generated (operation 208). The emotionalinformation can be stored in the storage unit 130, transmitted to acontent providing server, or additionally recorded in meta data of thecontent.

FIG. 3 is a view illustrating facial expression template informationaccording to an exemplary embodiment of the present invention.

Referring to FIG. 3, a plurality of facial expression templatesassociated with facial expressions representing neutrality, happiness,etc. are stored in the storage unit 130. Each facial expressioncorresponds to a type of emotion. Facial expression templatesrepresenting different strength with respect to each type of emotion canbe additionally stored. For example, a facial expression representinghappiness can be divided into 5 levels according to its strength.

FIG. 4 is a table in which the types of emotional information arelisted, according to an exemplary embodiment of the present invention.Referring to FIG. 4, the types of emotions 402 and the strength ofemotions 404 are illustrated.

FIG. 5 is a view illustrating emotional information generated accordingto an exemplary embodiment of the present invention. Referring to FIG.5, the emotional information includes types of emotional information 504for respective scenes and a representative emotion 502 of the content.In this exemplary embodiment, “sadness,” which is the representativeemotion of the content, is a user's emotion in four scenes, and is theemotion portrayed by a facial expression which the user most frequentlyhas while watching the content. The types of emotional information 504for each scene may be a representative emotion of emotional informationobtained by acquiring user images several times or periodically duringeach scene.

FIG. 6 is a flowchart of an emotional information generating methodaccording to another exemplary embodiment of the present invention,wherein generated emotional information is fed back to a contentprovider and used as meta data by the content provider.

Referring to FIG. 6, content is received from a content providing serverand is reproduced (operation 602). If a photographing signal is received(operation 604), a user image is acquired by photographing a user usinga camera, etc. installed in the emotional information generatingapparatus 100 illustrated in FIG. 1 (operation 606). The photographingsignal is generated at a predetermined time. For example, thephotographing signal is generated at a predetermined time designated inmeta data of the content, in units of a predetermined period, orwhenever a scene changes. By more frequently photographing the user, auser's emotion can be more exactly analyzed. Then, according to thefacial expression recognition method, the facial expression of the userimage is analyzed. In order to recognize the facial expression, first, afacial area is separated from the user image, and facial information isextracted from the facial area (operation 608). The extracted facialinformation is normalized (operation 610), and the normalized facialinformation is compared with facial expression template information, sothat the facial expression is analyzed and emotional informationcorresponding to the facial expression is generated (operation 612).Then, the emotional information is stored (operation 614). The processfrom operation 604 to operation 614 is repeated, until reproduction ofthe content is terminated (operation 616).

After the reproduction of the content is terminated, representativeemotional information, which represents the generated emotionalinformation, is generated (operation 618), and is transmitted to thecontent providing server (operation 620). According to applications, allthe generated emotional information and/or results obtained bystatistically processing or integrating the generated emotionalinformation, as well as the representative emotional information, can beprovided to the content providing server.

FIG. 7 is a view explaining a relationship between a content providerand a content receiver according to an exemplary embodiment of thepresent invention. In FIG. 7, the content receiver corresponds to theemotional information generating apparatus 100 illustrated in FIG. 1,and the content provider corresponds to a meta data generating apparatuscomprising a communication unit which receives emotional informationfrom the content receiver, and a meta data generating unit whichgenerates meta data for an emotion using the received emotionalinformation and couples the meta data to the content.

Referring to FIG. 7, the content provider generates content that is tobe serviced (operation 702), and transmits the content to the contentreceiver (operation 704). The content receiver acquires a user image(operation 708) while receiving and reproducing the content (operation706). The user's facial expression is recognized from the acquired userimage, and emotional information corresponding to the recognized facialexpression is generated (operation 710), and is transmitted to thecontent provider (operation 712). The content provider receives theemotional information from the content receiver (operation 714), andreflects the emotional information to the meta data of the content(operation 716). Then, a different content receiver which has receivedthe corresponding content can recognize people's response to the contentwith reference to the meta data.

The present invention can be applied to various types of contents. Inthe case of broadcast contents, since meta data is included in programdetails of an EPG, users' emotions for the broadcast contents can berecognized with reference to the EPG information. In the case of musiccontents, it is possible to photograph a user at a predetermined time orin a predetermined period by using a camera installed in a meta datagenerating apparatus at a content receiver side which reproduced themusic contents, recognize the user's facial expression, and generateemotional information corresponding to the user's facial expression. Inthe case of picture contents, it is possible to recognize a user'sfacial expression for each picture by using a camera installed in a metadata generating apparatus at a content receiver side which reproducesthe picture contents. The present invention can also be applied togames, visual telephone services, etc.

FIG. 8 is a flowchart illustrating a meta data generating methodaccording to an exemplary embodiment of the present invention.

Referring to FIG. 8, a meta data generating apparatus first receivesemotional information in relation to content from at least one clientsystem (operation 802). The at least one client system generates theemotional information for the content, while receiving and reproducingthe content from the meta data generating apparatus or a differentcontent providing server. The meta data generating apparatus generatesmeta data for an emotion using the received emotional information(operation 804). At this time, the emotional information received from aplurality of client systems is integrated. Then, a representative typeof emotion representing the content is determined, a representative typeof emotion which represents each scene of the content is determined, ora representative type of emotion for each predetermined time period isdetermined, so that meta data for an emotion including the determinedtype of emotion can be generated. As an exemplary method of determininga representative type of emotion from among a plurality of types ofemotion, as described above, a method of determining, as arepresentative type of emotion, a type of emotion which a user mostfrequently has can be used. The generated meta data for an emotion iscoupled to the content (operation 806). In order to couple the meta datato the content, a method of adding the meta data to existing meta dataof the content can be used. In the case when the content is a broadcastprogram, the meta data can be added to an EPG. In the case of content inwhich meta data is recorded in a form of a header, the meta data can beadded to header information. Before providing the content to the atleast one client system, the meta data generating apparatus can addinformation about a period or time when the client system obtains a userimage to the meta data of the content, in order to acquire emotionalinformation.

The meta data for the emotion coupled to the content can be applied tovarious fields, when the content is again used. The emotionalinformation included in the meta data can be used for publicizing thecontent, and the meta data for the emotion can be provided to a user sothat the user can select content. Also, only a part corresponding to auser's desired emotion is extracted from content so that he or she canwatch only a desired part, or new content including only the extractedpart can be generated.

The method according to the exemplary embodiments of the presentinvention as described above can also be embodied as computer readablecodes on a computer readable recording medium.

As described above, according to the exemplary embodiments of thepresent invention, it is possible to automatically acquire emotionalinformation by using a facial expression of a user who is appreciatingmultimedia content. Furthermore, it is possible to detect a change in auser's emotions according to respective scenes of image content, oracquire emotional information in relation to a specific part from whichusers' response can be analyzed.

By using the meta data generated according to the exemplary embodimentsof the present invention, a content provider can objectively recognizeusers' response to content and reflect the recognized information in themanufacturing of subsequent content or in users' response analysis.Also, users' emotional information in relation to content can be appliedto medical purposes.

Advertisers who advertise products by using content can easily becomeaware of advertising effects, and content users can easily search fordesired content with reference to content meta data including differentusers' emotional information. For example, by using emotionalinformation as a search keyword, users can search for movies which causelaughter in many people while they watch it, or search for and watchonly a part of a movie, which causes laughter.

While the present invention has been particularly shown and describedwith reference to exemplary embodiments thereof, it will be understoodby those of ordinary skill in the art that various changes in form anddetails may be made therein without departing from the spirit and scopeof the present invention as defined by the following claims.

What is claimed is:
 1. A method of reproducing content in a clientdevice, the method comprising: receiving, by the client device, thecontent with metadata corresponding to the content, from a server, themetadata comprising information about a representative type of emotioncorresponding to the content, wherein the information about therepresentative type of emotion is generated from emotional informationin relation to the content, and is recorded in an Electronic ProgramGuide (EPG) associated with the content, and wherein the emotionalinformation in relation to the content is obtained by extracting facialinformation from user images of one or more users who are provided withthe content; providing the information about the representative type ofemotion corresponding to the content, that is obtained from the metadatacorresponding to the content, to a user of the client device in aviewable format so that the user can reproduce portions of the contentby referring to the representative type of emotion in the EPG; andreproducing the content based on the information about therepresentative type of emotion.
 2. The method of claim 1, wherein themetadata is further recorded in a header of the content.
 3. Anon-transitory computer-readable recording medium storing a computerprogram for performing a method of reproducing content, the methodcomprising: receiving, by a client device, the content with metadatacorresponding to the content, from a server, the metadata comprisinginformation about a representative type of emotion corresponding to thecontent, wherein the information about the representative type ofemotion is generated from emotional information in relation to thecontent, and is recorded in an Electronic Program Guide (EPG) associatedwith the content, and wherein the emotional information in relation tothe content is obtained by extracting facial information from userimages of one or more users who are provided with the content; providingthe information about the representative type of emotion correspondingto the content, that is obtained from the metadata corresponding to thecontent, to a user of the client device in a viewable format so that theuser can reproduce portions of the content by referring to therepresentative type of emotion in the EPG; and reproducing the contentbased on the information about the representative type of emotion. 4.The non-transitory computer-readable recording medium of claim 3,wherein the metadata is further recorded in a header of the content. 5.An apparatus for reproducing content, the apparatus comprising: aprocessor configured to: receive, by a client device, the content withmetadata corresponding to the content, from a server, the metadatacomprising information about a representative type of emotioncorresponding to the content, wherein the information about therepresentative type of emotion is generated from emotional informationin relation to the content, and is recorded in an Electronic ProgramGuide (EPG) associated with the content, and wherein the emotionalinformation in relation to the content is obtained by extracting facialinformation from user images of one or more users who are provided withthe content, provide the information about the representative type ofemotion corresponding to the content, that is obtained from themetadata, to a user of the client device in a viewable format so thatthe user can reproduce portions of the content by referring to therepresentative type of emotion in the EPG, and reproduce the contentbased on the information about the representative type of emotion. 6.The apparatus of claim 5, wherein the metadata is further recorded in aheader of the content.
 7. A system comprising: a server comprising aprocessor configured to receive emotional information corresponding tocontent from at least one client, the emotional information beinggenerated by extracting facial information from user images of one ormore users who are provided with the content, generate information abouta representative type of emotion associated with the emotionalinformation, generate metadata corresponding to the content using theinformation about the representative type of emotion, and provide thecontent with the metadata comprising the information about therepresentative type of emotion corresponding to the content, wherein themetadata is recorded in an Electronic Program Guide (EPG) associatedwith the content; and a client device comprising a processor configuredto receive the content with the metadata corresponding to the content,from the server, provide the information about the representative typeof emotion corresponding to the content, that is obtained from themetadata, to a user of the client device in a viewable format so thatthe user can reproduce portions of the content by referring to therepresentative type of emotion in the EPG, and reproduce the contentbased on the information about the representative type of emotion. 8.The system of claim 7, wherein the metadata is further recorded in aheader of the content.